blm_tasks

BLM_tasks -- Blackbird Language Matrices -- aim to measure rule-like generalization in neural networks, through synthetic datasets that are generated based on combinations of rules relevant to particular targeted grammatical phenomena

Authors

• Paola Merlo Paola.Merlo@unige.ch
• Chunyang Jiang Chunyang.Jiang@unige.ch
• Giuseppe Samo Giuseppe.Samo@unige.ch
• Vivi Nastase vivi.a.nastase@gmail.com

Implementation

task.py implements:

• evaluate_predictions: to adjust the computation of the F1 score for the correct answer in our multiple choice setting
• format_example: to adjust mapping the input to the requested formatting

Usage

The adjusted evaluate_predictions method takes care of particularities of our set-up.

Checklist:

• [x ] I and my co-authors agree that, if this PR is merged, the code will be available under the same license as the genbench_cbt repository.
• [ x] Prior to submitting, I have ran the GenBench CBT test suite using the genbench-cli test-task tool.
• [ x] I have read the description of what should be in the doc.md of my task, and have added the required arguments.
• [ x] I have submitted or will submit an accompanying paper to the GenBench workshop.